Product suggestion and rules engine driven off of ancillary data

ABSTRACT

Curating ancillary data to be presented to audience members of a visual program content may include a) creating a timeline rule that correlates ancillary data objects to respective visual program content features, the visual program content features correlated to respective instances on a timeline of the visual program content, b) creating an environmental rule to correlate the ancillary data objects to respective environmental features of an audience member; and c) indicating that the ancillary data objects are to be presented to the audience member when both the timeline rule and the environmental rule are met such that the ancillary data objects may be presented to the audience member when both a) the respective ones of the visual program content features appear in the visual program content during playback by the audience member and b) the respective environmental features are present.

BACKGROUND

Media content is produced, processed, and then transmitted to consumers.In addition to traditional media content, the proliferation ofelectronic communication technologies has allowed for mass delivery ofdata related to or enhancing the content. For example, technologies suchas instant messaging provide a medium by which to deliver electronicinformation to one person or a large number of people very quickly.Electronic devices including, for example, personal computers, mobilephones, personal digital assistants, and television set-top boxes (e.g.,cable set top boxes, satellite set top boxes, etc.), provide readyaccess to consumers of information. The type and content of data thatmay be delivered via modern communication technologies varies greatlyand comprises everything from personal information, to informationalcontent, to advertisement.

At any moment in any piece of visual or audio-based content there is avast array of ancillary data that is virtually unlimited in itsassociation and context. Examples are not limited to, but include castmembers, location data, music, costume information, brands and products,and customary information native only to the individual programming. Butcurrent technologies are deficient in deploying such ancillary data forconsumption. Currently, there is not a way to adequately and efficientlycurate and distribute ancillary data relating to content.

SUMMARY OF THE INVENTION

The present disclosure provides methods and systems to address theseproblems. The present disclosure discloses a product suggestion andrules engine that allows a user to set conditions against this array ofdata. These conditions are vast in what can be applied but may bedesigned so as to establish rules and/or suggestions that would applyagainst any of the ancillary data. Ancillary data may then be triggeredbased on a combination of the assigned condition and the inheritance ofan externally detected conditional event. Some examples of this includethe ability to use costume data to reach out automatically against anarray of retailers so that a recommendation may be quickly establishedbased on the ancillary data assigned to that particular costume item.Further, the use of the conditionally defined rule, leverages thedetection of a user's or device's geographic data and even informationfrom their publicly available profile to trigger the event that appliesto the defined rules. This is so that the user may see the suggestedproduct from the retailer that best matches their geography and in a waythat appeals best to them based on their publicly available or alloweduser data.

Users have personal desires or inquiries about ancillary data withinprogramming. These personal desires or inquiries are often related tothe user's environment such as their geography, their gender, race,ethnicity, time zone, and other allowable and/or publicly availableprofile information. Rules or conditions may be assigned to anindividual piece of ancillary data to make good use of thisenvironmental information to specifically cater to audiences. On theother hand, where a retail product association may otherwise be offeredbut the state or geographic region has restrictions on such product, arule may avoid the offering. The invention may also help in fulfillinglaws, regulatory rules, and/or restrictions applicable to a particularpiece of data that may be restricted in terms of geography, age, orother reasons.

Prior to this invention, the ability did not exist to have rules thatcould be applied on a micro level within any moment in any regionrequested by a device or user against any class of ancillary data, orthat could leverage the viewer or devices detected factors andinstantaneously respond to that device or user based on the assignedrule and/or condition.

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various example systems, methods,and so on, that illustrate various example embodiments of aspects of theinvention. It will be appreciated that the illustrated elementboundaries (e.g., boxes, groups of boxes, or other shapes) in thefigures represent one example of the boundaries. One of ordinary skillin the art will appreciate that one element may be designed as multipleelements or that multiple elements may be designed as one element. Anelement shown as an internal component of another element may beimplemented as an external component and vice versa. Furthermore,elements may not be drawn to scale.

FIG. 1A illustrates a schematic diagram of an exemplary method or systemfor presenting ancillary data to audience members of a visual programcontent.

FIGS. 1B and 1C illustrate exemplary screen shots for presentingancillary data to audience members of a visual program content.

FIG. 2A illustrates an exemplary graphical user interface (GUI) forcurating ancillary data.

FIG. 2B illustrates a schematic diagram of an exemplary multi-timelinescenario for presentation of ancillary data.

FIGS. 3A-3G illustrate an exemplary graphical user interface (GUI) forcurating ancillary data.

FIG. 4 illustrates the exemplary GUI of FIG. 2A for curating ancillarydata.

FIG. 5 illustrates nine exemplary environmental rules or conditions.

FIGS. 6-10 illustrate block diagrams of exemplary rules to correlateancillary data objects to content features and/or environmental featuresof the audience member.

FIG. 11 illustrates a block diagram of an exemplary system for curatingand presenting ancillary data to be presented to audience members of thecontent.

FIG. 12 illustrates a flow diagram for an exemplary method for curatingancillary data to be presented to audience members of a visual programcontent.

FIG. 13 illustrates a flow diagram for an exemplary method forpresenting ancillary data to audience members of a visual programcontent.

FIG. 14 illustrates a block diagram of an exemplary machine for curatingand/or presenting ancillary data to be presented to audience members ofa visual program content.

DETAILED DESCRIPTION

FIG. 1A illustrates a schematic diagram of an exemplary method or systemfor presenting ancillary data to audience members of a visual programcontent. FIG. 1A shows an audiovisual content 1, which includes a visualportion 3 and an audio portion 5. The audiovisual content 1 may be amovie, a TV show, a sports event (e.g., basketball game), Internetvideo, a video game, a virtual reality (VR), augmented reality (AR), ormixed reality (MR) environment, or audio only programs via radio,Internet, etc.

FIG. 1A also shows ancillary data 7. The ancillary data 7 is data thatis related to the content and may include data describing the contentsuch as content name or content identification data, data about a scriptplayed out in the content, data about wardrobe wore by characters in thecontent, data including comments from performers, producers, ordirectors of the content, an Uniform Resource Locator (URL) to aresource that includes information about the content, data about musicin the audio of the content, the live score of a sports event, etc.Ancillary data 7 may include commercial data such as advertisement data.Ancillary data 7 may also include user data such as comments fromviewers of the content (e.g., twitter messages, etc.) Ancillary data 7may also include professional or technical data such as statistics ofthe content's audio including, for example, loudness or dynamic rangescaling of the content's audio, etc. Ancillary data 7 may also includeblockchain level access to another application. As can be seen from theabove examples, what constitutes ancillary data 7 may vary widely andmay be collected from a variety of sources. A user may select ancillarydata object (e.g., by clicking or touching the display on which thecontent (and/or the ancillary data object) is displayed) to interactwith the ancillary data. On example of interacting with ancillary datais obtaining information and/or purchasing a product suggested asancillary data. The system of FIG. 1A may present ancillary data 7synchronized to the content. 1. The ancillary data 7 may represent orlead to a product with which the audience member selecting the ancillarydata object may purchase or interact.

Some methods of synchronization of content 1 and ancillary data 7 may bestraightforward and may simply require an explicit data connectionbetween the content's source and the target or consumer. This explicittiming data communicates the timing to equipment at the consumerpremises. Other current methods of synchronization may rely on metadataattached to the content 1, which may or may not be present all the waythrough the signal chain from the source or content provider to thetarget or consumer since different facilities will use various workflowsor content container formats which may or may not support metadata.

FIG. 1A illustrates a representation 9 of the audio portion 5 of theaudiovisual content 1 in the form of an audio waveform signature. Therepresentation 9 matches the audio portion 5 of the audiovisual content1 at least to the extent that the audio portion 5 is identifiable fromthe representation 9 along the time t. In the embodiment of FIG. 1A, theancillary data 7 are each correlated to the representation 9 at instantsof the representation 9 corresponding to the instants of the audioportion 5 to which the ancillary data 7 is aligned. In this manner, theancillary data 7 may each be correlated to instants of the content 1.

In one embodiment, the ancillary data 7 may be correlated to a durationor time period (and not merely one instant) on the content 1. In suchembodiment, the ancillary data 7 may be correlated to two (or more)instants on the content 1 representing a start and an end, respectively,of the duration on the content 1 (e.g., a movie scene). In anotherembodiment, the ancillary data 7 may be correlated to start instant anda duration on the content 1 applicable to the ancillary data 7 defined.In such embodiment, the ancillary data 7 may be correlated to thestarting (or ending) instant on the content 1 representing a start andan end, respectively, of the duration on the content 1 (e.g., a moviescene) and the duration specified as an absolute term.

In the example of FIG. 1A, ancillary data A is correlated to the instantof representation 9 corresponding to time t_(x) of the audiovisualcontent 1. Ancillary data 7 b, 7 c, and 7 h are correlated to theinstant of representation 9 corresponding to time t_(x+1) of theaudiovisual content 1. Ancillary data 7 d and 7 e are correlated to theinstant of representation 9 corresponding to time t_(x+2). Ancillarydata 7 f is correlated to the instant of representation 9 correspondingto time t_(x+3) of the audiovisual content 1 and ancillary data 7 g iscorrelated to the instant of representation 9 corresponding to timet_(x+4) of the audiovisual content 1. Each of the ancillary data 7 andthe representation 9 may then be stored in a database that may be madeaccessible to future users or viewers of the audiovisual content 1. Thisway, when the audiovisual content 1 is distributed to those users orviewers, the representation 9 as well as the ancillary data 7 correlatedto the representation 9 may be available to those users or viewers.

At the user's premises, the audio portion 5 of the audiovisual content 1being received may be compared real-time to the representation 9 tosynchronize the audio portion 5 and hence the audiovisual content 1 tothe representation 9. Moreover, since the ancillary data 7 is correlatedto the instants of the representation 9 corresponding to theirrespective times of the audiovisual content 1, the ancillary data 7 maybe synchronized to the audiovisual content 1 even in the absence ofexplicit timing data.

Although in the embodiment of FIG. 1A the content's audio is used tosynchronize the ancillary data 7 to the content 1, the present inventionis not so limited. The ancillary data 7 may be synchronized to thecontent 1 as otherwise described herein and/or as known in the art.

In the illustrated embodiment of FIG. 1A, ancillary data 7 a is alignedto (i.e., it appears at or relates to) a time t_(x) of the audiovisualcontent 1. Ancillary data 7 b and 7 c appear at or relate to a timet_(x+1) of the audiovisual content 1. Ancillary data 7 d and 7 e appearat or relate to time t_(x+2). Ancillary data 7 f appears at or relatesto time t_(x+3) of the audiovisual content 1 and ancillary data 7 gappears at or relates to time t_(x+4). For example, ancillary data 7 aat t_(x) may indicate the content's name, True Blood, season 2, episode2. At time t_(x+1) (e.g., at 12 m 2 s) ancillary data 7 b describesSookie Stackhouse (character), played by Anna Paquin (actor) is wearingManolo Blahnik Hangisi 105 mm satin pump shoes (accessories) whileancillary data 7 c indicates that the music is Beethoven's MoonlightSonata performed by the London Symphony Orchestra. Ancillary data 7 dand 7 e may be twitter messages received at time t_(x+2) in which usersexpress their reactions to the audiovisual content 1 or a particularscene in the audiovisual content 1. Ancillary data 7 f may indicate achange at t_(x+3) in the prescribed loudness or dynamic range scaling ofthe content's audio due to a commercial break or can be made moregranular than just entire program. Ancillary data 7 g may indicate achange at t_(x+4) in the prescribed loudness or dynamic range scaling ofthe content's audio due to a return to the audiovisual content 1 fromthe commercial break.

Another type of ancillary data may be coordinate data of the visualportion 3 of the content 1. For example, ancillary data may include datathat identifies a set of coordinates representing a location within thevisual portion 3 of the audiovisual content 1 and data that identifiesthe center and shape of an object located within the visual portion 3 ofthe audiovisual content 1 at the location represented by the set ofcoordinates. In FIG. 1A, the ancillary data 7 h may be a set of x, ycoordinates (True Blood being a two-dimensional TV show) correspondingto the visual portion 3 of the content 1. The coordinates 7 h correspondto the location on the visual portion 3 of ancillary data 7 b, SookieStackhouse's Manolo Blahnik Hangisi 105 mm satin pump shoes or a relateditem as suggested by the system.

With this information being part of the ancillary data 7, a user mayquery the ancillary data system for audiovisual content in which ManoloBlahnik Hangisi 105 mm satin pump shoes appear. Search results can beManolo Blahnik, and/or 105 mm satin pumps (product). The result of thequery would be, not only True Blood, season 2, episode 2 as theaudiovisual content, but also t_(x+1) (e.g., at 12 m 2 s) as thetime+duration into the audiovisual content 1 in which the shoes appearand the coordinates x, y as the precise location of the shoes on thevisual portion 3. Alternatively, the user may query the ancillary datasystem for audiovisual content in which Manolo Blahnik Hangisi 105 mmsatin pump shoes appear at coordinates x, y. The result of the querywould be True Blood, season 2, episode 2 at time t_(x+1) (e.g., at 12 m2 s).

A user's actual geographical location may be detected and ancillary datarelating to the location may be offered to the user. For example, if amovie scene takes place at the user's current geographical location, themovie or scene may be offered to the user for viewing or purchase.

Similarly, with the coordinate set ancillary data available, a user mayquery the ancillary data system for what ancillary data is at a locationwithin the visual portion of an audiovisual content identified by a setof specific point, or shape coordinates. For example, the user maysearch for what is at coordinates x, y at time t_(x+1) or from within agiven shape of the audiovisual content 1, True Blood, season 2, episode2. A result of the query would be Manolo Blahnik Hangisi 105 mm satinpump shoes. To query the system the user may, for example, touch thescreen of a device at coordinates x, y at time t_(x+1) of theaudiovisual content 1, True Blood, season 2, episode 2. The system maydetect the touch at the specific location, search ancillary data, andoutput information identifying the object(s) at the specific location.

The above query combinations are merely illustrative. Many other querycombinations are possible in which coordinates as ancillary data allowfor more advanced utilization of the audiovisual content and ancillarydata in general.

Also, FIG. 1A illustrates a two-dimensional example (True Blood being atwo-dimensional TV show) but the ancillary data system disclosed here isnot limited to two dimensions and may include three-dimensionalcoordinates (x, y, z) for three-dimensional content (e.g., 3D videogames, 3D movies, 3D virtual reality, etc.) as described below.

The inclusion of coordinates as ancillary data provides furtheropportunities for more advanced utilization of the audiovisual contentand ancillary data in general. Coordinates as ancillary data maycorrespond to simple relative coordinates such as, for example,coordinates x, y representing simply the location within a video frame(e.g., x=0-1920, y=0-1080) or a given shape (min of three coordinates ifaspect ratio is known) of the content 1. However, coordinates asancillary data may correspond to coordinates relative to alternativespaces or areas such as, for example, coordinates x, y representing thelocation within the video frame of the content 1 and at the same timethe location within another space or area (e.g., a virtual space, aspace within a video game, a space within a different audiovisualcontent, etc.) Coordinates as ancillary data may also correspond toabsolute coordinates that can be correlated to other spaces or areassuch as, for example, coordinates x, y representing the location withinthe video frame of the content 1 and at the same time the locationwithin a real world space (e.g., a stadium, a city, a country, a planet,the universe, etc.)

Moreover, coordinates corresponding to alternative spaces or areas donot need to be absolute or relative to the coordinates corresponding tothe location on the visual portion 3 of the content 1. These coordinatescorresponding to alternative spaces or areas may simply be tied orcorrelated to the coordinates corresponding to the location on thevisual portion 3 of the content 1. For example, the coordinatescorresponding to alternative spaces or areas may be correlated asancillary data to the corresponding instant in the synchronization data9 to tie or correlate them to the coordinates corresponding to thelocation on the visual portion 3 of the content 1. These additionallayer of coordinates become an additional layer of ancillary data.

FIGS. 1B and 1C illustrate exemplary screen shots for presentingancillary data to audience members of a visual program content. Thecontent 1 in the example of FIGS. 1B and 1C is the movie Ocean's 8.

In FIG. 1B, while enjoying the content 1 an audience member has touchedor clicked on the character Debbie Ocean played by Sandra Bullock.Ancillary data correlated to that instant in the timeline (e.g., thewhole movie, the particular scene, the particular character, etc.) andthat location on the screen becomes visible on the screen. In theexample of FIG. 1B, the audience member is presented with the options ofaccessing information on the actress, Ms. Bullock, the jacket she iswearing (Burberry®), or the glasses in front of her (Cartier®). Theaudience member may further touch or click on the choices presented tobe given the opportunity to learn more about Ms. Bullock and/or purchasethe jacket or the glasses.

In FIG. 1C, the audience member has touched or clicked on the name ofthe restaurant, Veselka, on the window. As a result, ancillary datacorrelated to that instant in the timeline (e.g., the whole movie, theparticular scene, the particular character, etc.) and that location onthe screen becomes visible on the screen. In the example of FIG. 1C, theaudience member is presented with the options of accessing informationon the restaurant, Veselka, or the geographical location (e.g., Google®Maps) of the restaurant. The audience member may further touch or clickon the choices presented to be given the opportunity to learn more aboutthese places and maybe purchase a gift certificate to the restaurant.

The question becomes: how does the ancillary data 7 become correlated tospecific instants of the content 1?

FIG. 2A illustrates an exemplary graphical user interface (GUI) 200 forcurating ancillary data. The exemplary GUI 200 includes six tracks V1-V6and timing information in the form of a time code TC1 and an edge codeEC1. Aspects of the content 1 such as content title, characters, contentscript or plot, content scene, wardrobe wore by the characters,performers, producers, or directors of the content, items appearing inthe content, geographical locations where the content storyline takesplace, music in audio of the content, etc. may be defined as contentfeatures and may be correlated in the GUI 200 to respective instances onthe timeline TC1 of the visual program content 1. For example, track V6spans the length of the whole program or content 1 but track V5 includesscenes that span only the length of a particular scene within thecontent 1. Thus, scene 1 and scene 2 are content features that arecorrelated to respective instances or time period of the content 1.Similarly, track V4 defines time periods within the content 1 in which acontent feature, a particular character “Susan Jones,” appears. Track V3defines time periods within the content 1 in which another particularcontent feature, character “John Doe,” appears. Track V2 defines timeperiods within the content 1 in which a particular wardrobe “BlackButton Dress Shirt” appears. Track V3 defines time periods within thecontent 1 in which another particular wardrobe “Nike Running Shoes”appears. Although a number of tracks is used for purposes ofillustration, a potentially infinite number of tracks may be possible toallow for, for example, many actors, wardrobe, etc.

Although one timeline is shown in FIG. 2A for purposes of illustration,a potentially infinite number of timelines are possible. For example,each user may have a different timeline, or a different timeline may beused depending of the outlet for the content 1. If the content 1 isplaying on a first channel or network, one timeline may be used while,if the content 1 is playing on a second (different from the first)channel or network, a second (different from the one) timeline may beused. FIG. 2B illustrates an example of this multiple timeline approach.After rule 1 is executed, the system may diverge from timeline 1 basedon some event taking place (or not taking place). So, for example, ifthe audience member has not engaged for some set period of time, thesystem may present new interactions by switching to a timeline 2. If,instead, the system had continued on timeline 1, the system may branchto yet more different timelines based on conditions such asenvironmental rules as described herein. So, in FIG. 2B, upon rule 2taking effect, the timeline diverts from timeline 1. Upon rule 3 takingeffect, the timeline diverts in yet another direction. Ultimately, alarge number of timelines is possible. In the example of FIG. 2Btimelines 3, 4, 5, and 6 may be engaged depending on conditions.

The visual program content 1 features being correlated to respectiveinstances on the timeline of the visual program content 1 may allow forthe correlation of ancillary data 7 to the timeline by creating timelinerules that refer to the content features. As described in detail below,the GUI 200 may be used to reflect timeline rules that correlateancillary data objects to program features of the content 1. As alsodescribed in detail below, the GUI 200 may further be used to reflectenvironmental rules that correlate ancillary data objects toenvironmental features of an audience member.

FIGS. 3A-3G illustrate an exemplary graphical user interface (GUI) 300for curating ancillary data. The GUI 300 may be used for curatingancillary data by creating timeline rules that correlate ancillary dataobjects to content features and environmental rules that correlateancillary data objects to environmental features of an audience member.In the example of FIGS. 3A-3G, the GUI 300 is used to create a rule thatcorrelates ancillary data regarding a product (a bottle of 1961 ChateauChavel Blanc wine) to time periods in the content 1 in which a castmember appears in the content 1 and to environmental features of theaudience member.

As shown in FIG. 3A, a user or curator may find the product (a bottle of1961 Chateau Chavel Blanc wine) including information (e.g., name,description, brand name, display link, shopping URL, etc.) about theproduct as an ancillary data object in a database. The user or curatormay then add a timeline rule in which the ancillary data object (in thiscase the wine bottle) is correlated to the content feature by clickingNEW RULE.

The wine bottle is an example where a specific product may be offered asancillary data. In one embodiment, instead of a specific product, aspecific service or shopping engine (e.g., Amazon®, shop.com) may beoffered as ancillary data in much the same way. Services and/or shoppingengines are offered in a similar manner as products, however when a userselects an ancillary data object (e.g., by clicking or touching thedisplay on which the content (and/or the ancillary data object) isdisplayed) for a service and/or shopping engine, the rules engine mayuse keywords from the content 1 or content feature (e.g., a particularcharacter's name correlated to the ancillary data object or any otheruser defined keywords) to create a query that may be sent to the serviceor shopping engine to return product results, a product suggestion ineffect.

FIG. 3B illustrates how the user or curator may add a new timeline rulein which the ancillary data object (the wine bottle) is correlated to acontent feature (a cast member) using the GUI 300. The user may indicatethe timeline rule for the ancillary data object (the wine bottle) as:cast members>specific>equals>Ana de Armas. If this simple timeline ruleis implemented, every time Ana de Armas would appear on theprogram/content 1, the ancillary data object referring to the bottle of1961 Chateau Chavel Blanc wine would be presented to the audience memberwatching the content 1.

Rules stacking is possible. The user or curator may continue to addtimeline rules in which the ancillary data object (the wine bottle) iscorrelated to other content features. For example, the user may add atimeline rule that specifies that the ancillary data object is to bepresented to the audience member “Any time Ana de Armas is on screen ANDthe mood is happy AND it's day time on screen.” Timeline rules canbecome very complex and utilize Boolean logic with grouping to allowcreation of theoretically any possible combination of data to the rule.

The GUI 300 may further be used for curating ancillary data by creatingenvironmental rules that correlate ancillary data objects toenvironmental features of the audience member. Environmental features ofthe audience member may include a date range in which the audiencemember watches the visual program content, a geographical location ofthe audience member, gender of the audience member, age range of theaudience member, etc.

As illustrated in FIG. 3C, a user or curator may add an environmentalrule consisting of viewing conditions by clicking ADD CONDITION. Asshown in FIG. 3D, the environmental rule portion or new condition mayinvolve a date range for when the audience member consumes the content1, an age range of the audience member, gender of the audience member,or geography of the audience member. The new condition or environmentalrule portion may also be an invert condition in which, instead ofspecifying a date range, an age range, gender, or geography of theaudience member, the opposite is specified. So, for example, everygeography except China, any age except 1-17, any day except September11, etc. may be specified.

FIG. 3E illustrates that a date range condition was selected andspecified as from Sep. 11, 2018 to Dec. 31, 2018. The user or curatormay continue to stack conditions as desired to create the desiredenvironmental rule or rules. FIG. 3F illustrates the final environmentalrule specifying the target audience member as a female, ages 18-52, fromMassachusetts watching the content 1 from Sep. 11, 2018 to Dec. 31,2018. As shown in FIG. 3G, a rule has been created that ancillary dataregarding the product (a bottle of 1961 Chateau Chavel Blanc wine) is tobe presented to an audience member that is a female, ages 18-52, fromMassachusetts watching the content 1 from Sep. 11, 2018 to Dec. 31, 2018at time periods in the content 1 in which a cast member “Ana de Armas”appears in the content 1. The ancillary data object will be presented toparticular audience members accordingly.

FIG. 4 illustrates the exemplary GUI 200 for curating ancillary data. Asdescribed above, the GUI 200 may be used to reflect timeline rules thatcorrelate ancillary data objects to program features of the content 1and environmental rules that correlate ancillary data objects toenvironmental features of an audience member. The rules may berepresented by icons. For example, a rule R6 has been applied to thecontent feature (the whole content 1) on track V6 which spans the lengthof the whole program or content 1. A rule R4 has been applied to thecontent feature Scene 1 which spans the length of the particular sceneon track V5 while rule R5 has been applied to the content feature Scene2 which spans the length of the particular scene on track V5. Similarly,a rule R1 has been applied to the content feature, character “SusanJones,” on track V4. A rule R8 has been applied to the content feature,character “John Doe,” on track V3 whenever John Doe appears. Incontrast, a rule R2 has been set for a specific first time period (notall time periods) within the content 1 in which the content feature,character “John Doe,” appears in the content 1 and a rule R3 has beenset for a specific second time period (not all time periods) within thecontent 1 in which the content feature, character “John Doe,” appears inthe content 1.

As shown in FIG. 5, there may be a preset list of common environmentalrules or conditions as well as the ability for the user to create customrules. FIG. 5 lists nine exemplary common environmental rules orconditions. Rule 1 (R1)=female audience member, rule 2 (R2)=maleaudience member, rule 3 (R3)=audience member geography is China, rule 4(R4)=geo active, rule 5 (R5)=audience member watching from August toOctober 2017, rule 6 (R6)=audience member in the United States, rule 7(R7)=audience member geography is not China, rule 8 (R8)=audiencemember's age is 18-35, and rule 9 (R9)=audience member's age is 35-50.

Returning to FIG. 4, let's study the rules applied to the contentfeature, character “John Doe,” on track V3. First, the rule R6 (audiencemember in the United States) applies to the whole content 1. Second, therule R4 (geo active) applies to the time periods in which “John Doe”appears in Scene 1 while rule R5 (audience member watching from Augustto October 2017) applies to the time periods in which “John Doe” appearsin Scene 2. The rule R8 (audience member's age is 18-35) has been setfor whenever John Doe appears. In contrast, a rule R2 (male audiencemember) has been set for specific time periods (not all time periods)within the content 1 in which the content feature, character “John Doe,”appears and a rule R3 (audience member geography is China) has been setfor other specific time periods (not all time periods) within thecontent 1 in which the content feature, character “John Doe,” appears.

In one embodiment, rules are applied top to bottom so that program rulesare applied first, scene rules second, track rules third, and featurerules last. In the example of FIG. 4, John Doe appears three times. Thefirst time John Doe appears, rules R6, R4, R8, and R2 (in that order)apply. Therefore, the ancillary data object is presented to the audiencemember if the audience member in the United States, the audience memberis geo active, the audience member is 18-35, and the audience member ismale. The second time John Doe appears, rules R6, R4, and R8 (in thatorder) apply. Therefore, the ancillary data object is presented to theaudience member if the audience member is in the United States, theaudience member is geo active, and the audience member is 18-35 (male orfemale). The third time John Doe appears, rules R6, R5, R8, and R3 (inthat order) apply. Therefore, the ancillary data object is presented tothe audience member if the audience member in the United States, theaudience member is watching from August to October 2017, and theaudience member is 18-35. The ancillary data object is also presented toall audience members in China because of rule R3.

It should be clear from the above that curation may take place post hocbut also live. For example, ancillary data may include live scores of abasketball game as the game is being played. Curation may be effected bya human user or operator or by a machine operating based on analgorithm, artificial intelligence, etc.

FIGS. 6-10 illustrate block diagrams of exemplary rules to correlateancillary data objects to content features and/or environmental featuresof the audience member.

FIG. 6 illustrates a simple timeline rule 60 applicable to a program 1.In the timeline rule 60, if a particular character is currently presenton screen, an experience A (which presents one or more ancillary dataobjects to the audience member) is offered.

FIG. 7 illustrates a more complex timeline rule 61 applicable to theprogram 1. In the timeline rule 61, if a particular character iscurrently present on screen, and the movie takes place at a specificlocation, and a second character appears with the first character formore than 30 seconds, an experience A (which presents one or moreancillary data objects to the audience member) is offered.

FIG. 8 illustrates a simple environmental rule 62 applicable to theprogram 1. In the environmental rule 62, if the audience member is male,an experience A (which presents one or more ancillary data objects tothe audience member) is offered while if the audience member is female,an experience B (which presents one or more ancillary data objectsdifferent from those of experience A to the audience member) is offered.

FIG. 9 illustrates a more complex environmental rule 63 applicable tothe program 1. In the environmental rule 63, if the audience member ismale, older than 30, in Mexico, where it is sunny, an experience Y(which presents one or more ancillary data objects to the audiencemember) is offered until a certain date after which an experience X(which presents one or more ancillary data objects different from thoseof experience Y to the audience member) is offered.

FIG. 10 illustrates a combination environmental and timeline rule 64applicable to the program 1. In the combo rule 64, if a particularcharacter is currently present on screen, and the movie takes place at aspecific location, and a second character appears with the firstcharacter for more than 30 seconds, AND if the audience member is male,older than 30, in Mexico, where it is sunny, an experience Y (whichpresents one or more ancillary data objects to the audience member) isoffered until a certain date after which an experience X (which presentsone or more ancillary data objects different from those of experience Yto the audience member) is offered.

FIG. 11 illustrates a block diagram of an exemplary system 10 forcurating and presenting ancillary data to be presented to audiencemembers of the content 1. The system 10 includes three major components:the content distributor 20, the consumer 30, and the ancillary datalocation 40. FIG. 11 also shows the medium M through which the contentdistributor 20, the consumer 30, and the ancillary data location 40communicate with each other.

The element 20 is not limited to broadcasters or broadcasting facilitiesor equipment. In practice, the content distributor 20 may represent anyfacility or equipment that is part of or used in preproduction,production, postproduction, quality control, mastering equipment,broadcasting of any type (including professional or social mediabroadcasting), or other method of sending and distributing audio visualcontent, that touches the audiovisual content 1 prior to and duringplayout for transmission or broadcasting including a consumer's owndevices such as a DVR, natural language processing devices (e.g.,Amazon's Echo), etc.

Similarly, although for ease of explanation the present disclosurerefers to the element 30 as the consumer 30, the element 30 is notlimited to consumers or consumer premises or equipment. In practice, theconsumer 30 may represent any premise or equipment that touches theaudiovisual content 1 during or post playout for transmission,broadcasting, etc.

Regarding the ancillary data location 40 and authorship or collection ofancillary data 7, ancillary data 7 may be obtained or collected prior toplayout, broadcast, distribution or performance of the audiovisualcontent 1. For example, ancillary data 7 may be obtained or collectedduring preproduction, production, post-production, quality control, ormastering of the audiovisual content 1. Ancillary data 7 may also beobtained or collected during playout, broadcast, distribution orperformance of the audiovisual content 1. For example, if theaudiovisual content 1 is a TV show, ancillary data 7 may be obtained orcollected during a first or subsequent broadcast of the TV show.

Regarding storage and distribution, ancillary data 7 collected may bestored in a database 44 that may be made accessible to future users orviewers of the audiovisual content 1. This way, when the audiovisualcontent 1 is later distributed to those users or viewers, the ancillarydata 7 may be available to those users or viewers for consumption at thesame time as the audiovisual content 1. The ancillary data 7 appears ormanifests itself aligned in time to the audiovisual content 1.

Also, the medium M may be any medium used to transmit content 1 or datagenerally such as, for example, the Internet, satellite communication,radio communication, television communication (broadcast or cable), etc.Although in the figures the medium M is shown as being shared by thecontent distributor 20, the consumer 30, and the ancillary data location40, communication between these elements does not need to take place inthe same medium. So, for example, the content distributor 20 maycommunicate with the consumer 30 via satellite while the contentdistributor 20 communicates to the ancillary data location 40 via theInternet.

In the example of FIG. 11, the content distributor 20 transmits theaudiovisual content 1 to the consumer 30. The consumer 30 receives theaudiovisual content 1 from the content distributor 20 and the ancillarydata 7 from the ancillary data location 40.

The ancillary data location 40 may include a machine or group ofmachines for curating and presenting ancillary data correlated tocontent 1. The ancillary data location 40 may include a transceiver 42that communicates (i.e., transmits and receives) the content 1 and/orthe ancillary data 7 and a processor 43 that correlates the ancillarydata 7 to the content 1 by creating timeline and environmental rules asdescribed above. The ancillary data location 40 may also include adatabase 44 that stores the ancillary data 7 correlated to instants inthe content 1.

The ancillary data location 40 may be a location accessible to thecontent distributor 20 and the consumer 30, such as the cloud or a localarchive with general accessibility (e.g., via a link as described above)that may be controlled by subscription, password, etc.

The system 10 may be implemented using software, hardware, analog ordigital techniques.

Exemplary methods may be better appreciated with reference to the flowdiagrams of FIGS. 12 and 13. While for purposes of simplicity ofexplanation, the illustrated methodologies are shown and described as aseries of blocks, it is to be appreciated that the methodologies are notlimited by the order of the blocks, as some blocks can occur indifferent orders or concurrently with other blocks from that shown anddescribed. Moreover, less than all the illustrated blocks may berequired to implement an exemplary methodology. Furthermore, additionalmethodologies, alternative methodologies, or both can employ additionalblocks, not illustrated.

In the flow diagrams, blocks denote “processing blocks” that may beimplemented with logic. The processing blocks may represent a methodstep or an apparatus element for performing the method step. The flowdiagrams do not depict syntax for any particular programming language,methodology, or style (e.g., procedural, object-oriented). Rather, theflow diagrams illustrate functional information one skilled in the artmay employ to develop logic to perform the illustrated processing. Itwill be appreciated that in some examples, program elements liketemporary variables, routine loops, and so on, are not shown. It will befurther appreciated that electronic and software applications mayinvolve dynamic and flexible processes so that the illustrated blockscan be performed in other sequences that are different from those shownor that blocks may be combined or separated into multiple components. Itwill be appreciated that the processes may be implemented using variousprogramming approaches like machine language, procedural, objectoriented or artificial intelligence techniques.

FIG. 12 illustrates a flow diagram for an exemplary method 600 forcurating ancillary data to be presented to audience members of a visualprogram content.

The method 600 includes at 610 creating a timeline rule that correlatesancillary data objects to respective visual program content features.The visual program content features are correlated to respectiveinstances on a timeline of the visual program content.

At 620, the method 600 further includes creating an environmental ruleto correlate the ancillary data objects to respective environmentalfeatures of an audience member.

At 630, the method 600 further includes indicating that the ancillarydata objects are to be presented to the audience member when both thetimeline rule and the environmental rule are met such that the ancillarydata objects may be presented to the audience member when both a) therespective ones of the visual program content features appear in thevisual program content during playback by the audience member and b) therespective environmental features are present.

FIG. 13 illustrates a flow diagram for an exemplary method 700 forpresenting ancillary data to audience members of a visual programcontent.

The method 700 includes at 710 receiving ancillary data objects that arecorrelated to a) respective visual program content features of thevisual program content and b) respective environmental features of anaudience member of the visual program content. The visual programcontent features are correlated to respective instances on a timeline ofthe visual program content.

At 720, the method 700 further includes presenting the ancillary dataobjects to the audience member when a) the respective visual programcontent features appear in the visual program content during playback bythe audience member and b) the respective environmental features arepresent.

While the figures illustrate various actions occurring in serial, it isto be appreciated that various actions illustrated could occursubstantially in parallel, and while actions may be shown occurring inparallel, it is to be appreciated that these actions could occursubstantially in series. While a number of processes are described inrelation to the illustrated methods, it is to be appreciated that agreater or lesser number of processes could be employed, and thatlightweight processes, regular processes, threads, and other approachescould be employed. It is to be appreciated that other exemplary methodsmay, in some cases, also include actions that occur substantially inparallel. The illustrated exemplary methods and other embodiments mayoperate in real-time, faster than real-time in a software or hardware orhybrid software/hardware implementation, or slower than real time in asoftware or hardware or hybrid software/hardware implementation.

FIG. 14 illustrates a block diagram of an exemplary machine 800 forcurating and/or presenting ancillary data to be presented to audiencemembers of a visual program content. The machine 800 includes aprocessor 43, a memory 804, and I/O Ports 810 operably connected by abus 808.

In one example, the machine 800 may receive input signals including theaudiovisual content 1, the visual portion 3, the audio portion 5, theancillary data 7, the representation 9, etc. via, for example, I/O Ports810 or I/O Interfaces 818. The machine 800 may also include thetransceiver 42, the processor 43, and the database 44 of the ancillarydata location 40. Thus, the content distributor 20, the consumer 30, orthe ancillary data location 40 may be implemented in machine 800 ashardware, firmware, software, or a combination thereof and, thus, themachine 800 and its components may provide means for performingfunctions described and/or claimed herein as performed by thetransceiver 42, the processor 43, and the database 44.

The processor 43 can be a variety of various processors including dualmicroprocessor and other multi-processor architectures. The memory 804can include volatile memory or non-volatile memory. The non-volatilememory can include, but is not limited to, ROM, PROM, EPROM, EEPROM, andthe like. Volatile memory can include, for example, RAM, synchronous RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM).

A disk 806 may be operably connected to the machine 800 via, forexample, an I/O Interfaces (e.g., card, device) 818 and an I/O Ports810. The disk 806 can include, but is not limited to, devices like amagnetic disk drive, a solid-state disk drive, a floppy disk drive, atape drive, a Zip drive, a flash memory card, or a memory stick.Furthermore, the disk 806 can include optical drives like a CD-ROM, a CDrecordable drive (CD-R drive), a CD rewriteable drive (CD-RW drive), ora digital video ROM drive (DVD ROM). The memory 804 can store processes814 or data 816, for example. The disk 806 or memory 804 can store anoperating system that controls and allocates resources of the machine800.

The bus 808 can be a single internal bus interconnect architecture orother bus or mesh architectures. While a single bus is illustrated, itis to be appreciated that machine 800 may communicate with variousdevices, logics, and peripherals using other busses that are notillustrated (e.g., PCIE, SATA, Infiniband, 1394, USB, Ethernet). The bus808 can be of a variety of types including, but not limited to, a memorybus or memory controller, a peripheral bus or external bus, a crossbarswitch, or a local bus. The local bus can be of varieties including, butnot limited to, an industrial standard architecture (ISA) bus, amicrochannel architecture (MCA) bus, an extended ISA (EISA) bus, aperipheral component interconnect (PCI) bus, a universal serial (USB)bus, and a small computer systems interface (SCSI) bus.

The machine 800 may interact with input/output devices via I/OInterfaces 818 and I/O Ports 810. Input/output devices can include, butare not limited to, a keyboard, a microphone, a pointing and selectiondevice, cameras, video cards, displays, disk 806, network devices 820,and the like. The I/O Ports 810 can include but are not limited to,serial ports, parallel ports, and USB ports.

The machine 800 can operate in a network environment and thus may beconnected to network devices 820 via the I/O Interfaces 818, or the I/OPorts 810. Through the network devices 820, the machine 800 may interactwith a network. Through the network, the machine 800 may be logicallyconnected to remote computers. The networks with which the machine 800may interact include, but are not limited to, a local area network(LAN), a wide area network (WAN), and other networks. The networkdevices 820 can connect to LAN technologies including, but not limitedto, fiber distributed data interface (FDDI), copper distributed datainterface (CDDI), Ethernet (IEEE 802.3), token ring (IEEE 802.5),wireless computer communication (IEEE 802.11), Bluetooth (IEEE802.15.1), Zigbee (IEEE 802.15.4) and the like. Similarly, the networkdevices 820 can connect to WAN technologies including, but not limitedto, point to point links, circuit switching networks like integratedservices digital networks (ISDN), packet switching networks, and digitalsubscriber lines (DSL). While individual network types are described, itis to be appreciated that communications via, over, or through a networkmay include combinations and mixtures of communications.

Definitions

The following includes definitions of selected terms employed herein.The definitions include various examples or forms of components thatfall within the scope of a term and that may be used for implementation.The examples are not intended to be limiting. Both singular and pluralforms of terms may be within the definitions.

“Content” corresponds to still images, segments of audio media, videomedia, or audio/visual (AV) media and include information that isembodied, stored, transmitted, received, processed, or otherwise usedwith at least one medium. Common media content formats include FLVformat (flash video), Windows Media Video, RealMedia, MFX, Quicktime,MPEG, MP3, DivX, JPEGs, and Bitmaps. As used herein, the terms “mediaclips”, “media content,” “information content,” and “content” may beused interchangeably.

“Data store” or “database,” as used herein, refers to a physical orlogical entity that can store data. A data store may be, for example, adatabase, a table, a file, a list, a queue, a heap, a memory, aregister, and so on. A data store may reside in one logical or physicalentity or may be distributed between two or more logical or physicalentities.

“Logic,” as used herein, includes but is not limited to hardware,firmware, software or combinations of each to perform a function(s) oran action(s), or to cause a function or action from another logic,method, or system. For example, based on a desired application or needs,logic may include a software-controlled microprocessor, discrete logiclike an application specific integrated circuit (ASIC), a programmedlogic device, a memory device containing instructions, or the like.Logic may include one or more gates, combinations of gates, or othercircuit components. Logic may also be fully embodied as software. Wheremultiple logical logics are described, it may be possible to incorporatethe multiple logical logics into one physical logic. Similarly, where asingle logical logic is described, it may be possible to distribute thatsingle logical logic between multiple physical logics.

An “operable connection,” or a connection by which entities are“operably connected,” is one in which signals, physical communications,or logical communications may be sent or received. Typically, anoperable connection includes a physical interface, an electricalinterface, or a data interface, but it is to be noted that an operableconnection may include differing combinations of these or other types ofconnections sufficient to allow operable control. For example, twoentities can be operably connected by being able to communicate signalsto each other directly or through one or more intermediate entities likea processor, operating system, a logic, software, or other entity.Logical or physical communication channels can be used to create anoperable connection.

In broadcasting, “playout” is a term for the transmission of radio or TVchannels from the broadcaster into broadcast networks that delivers thecontent to the audience.

“Signal,” as used herein, includes but is not limited to one or moreelectrical or optical signals, analog or digital signals, data, one ormore computer or processor instructions, messages, a bit or bit stream,or other means that can be received, transmitted, or detected.

“Software,” as used herein, includes but is not limited to, one or morecomputer or processor instructions that can be read, interpreted,compiled, or executed and that cause a computer, processor, or otherelectronic device to perform functions, actions or behave in a desiredmanner. The instructions may be embodied in various forms like routines,algorithms, modules, methods, threads, or programs including separateapplications or code from dynamically or statically linked libraries.Software may also be implemented in a variety of executable or loadableforms including, but not limited to, a stand-alone program, a functioncall (local or remote), a servlet, an applet, instructions stored in amemory, part of an operating system or other types of executableinstructions. It will be appreciated by one of ordinary skill in the artthat the form of software may depend, for example, on requirements of adesired application, the environment in which it runs, or the desires ofa designer/programmer or the like. It will also be appreciated thatcomputer-readable or executable instructions can be located in one logicor distributed between two or more communicating, co-operating, orparallel processing logics and thus can be loaded or executed in serial,parallel, massively parallel and other manners.

Suitable software for implementing the various components of the examplesystems and methods described herein may be produced using programminglanguages and tools like Java, Pascal, C#, C++, C, CGI, Perl, SQL, APIs,SDKs, assembly, firmware, microcode, or other languages and tools.Software, whether an entire system or a component of a system, may beembodied as an article of manufacture and maintained or provided as partof a computer-readable medium as defined previously. Another form of thesoftware may include signals that transmit program code of the softwareto a recipient over a network or other communication medium. Thus, inone example, a computer-readable medium has a form of signals thatrepresent the software/firmware as it is downloaded from a web server toa user. In another example, the computer-readable medium has a form ofthe software/firmware as it is maintained on the web server. Other formsmay also be used.

“User” or “consumer,” as used herein, includes but is not limited to oneor more persons, software, computers or other devices, or combinationsof these.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a memory. These algorithmic descriptions and representationsare the means used by those skilled in the art to convey the substanceof their work to others. An algorithm is here, and generally, conceivedto be a sequence of operations that produce a result. The operations mayinclude physical manipulations of physical quantities. Usually, thoughnot necessarily, the physical quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a logic and the like.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like. It should be borne in mind,however, that these and similar terms are to be associated with theappropriate physical quantities and are merely convenient labels appliedto these quantities. Unless specifically stated otherwise, it isappreciated that throughout the description, terms like processing,computing, calculating, determining, displaying, or the like, refer toactions and processes of a computer system, logic, processor, or similarelectronic device that manipulates and transforms data represented asphysical (electronic) quantities.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim. Furthermore, to the extentthat the term “or” is employed in the detailed description or claims(e.g., A or B) it is intended to mean “A or B or both”. When theapplicants intend to indicate “only A or B but not both” then the term“only A or B but not both” will be employed. Thus, use of the term “or”herein is the inclusive, and not the exclusive use. See, Bryan A.Garner, A Dictionary of Modern Legal Usage 624 (2d. Ed. 1995).

While example systems, methods, and so on, have been illustrated bydescribing examples, and while the examples have been described inconsiderable detail, it is not the intention of the applicants torestrict or in any way limit scope to such detail. It is, of course, notpossible to describe every conceivable combination of components ormethodologies for purposes of describing the systems, methods, and soon, described herein. Additional advantages and modifications willreadily appear to those skilled in the art. Therefore, the invention isnot limited to the specific details, the representative apparatus, andillustrative examples shown and described. Thus, this application isintended to embrace alterations, modifications, and variations that fallwithin the scope of the appended claims. Furthermore, the precedingdescription is not meant to limit the scope of the invention. Rather,the scope of the invention is to be determined by the appended claimsand their equivalents.

What is claimed is:
 1. A machine or group of machines for curatingancillary data to be presented to audience members of a visual programcontent, comprising: a processor configured to create: a timeline rulethat correlates ancillary data objects to respective visual programcontent features, the visual program content features correlated torespective instances on a timeline of the visual program content; and anenvironmental rule that correlates the ancillary data objects torespective environmental features of an audience member; and theprocessor configured to transmit a signal indicating that the ancillarydata objects are to be presented to the audience member when both thetimeline rule and the environmental rule are met such that the ancillarydata objects may be presented to the audience member when both a) therespective ones of the visual program content features appear in thevisual program content during playback by the audience member and b) therespective environmental features are present.
 2. The machine or groupof machines of claim 1, wherein the visual program content features areselected from a group comprising: content title; characters; contentscript or plot; content scene; wardrobe wore by the characters;performers, producers, or directors of the content; items appearing inthe content; geographical locations where the content storyline takesplace; music in audio of the content; and statistics related to thecontent such as sports statistics.
 3. The machine or group of machinesof claim 1, wherein the visual program content features being correlatedto respective instances on a timeline of the visual program contentincludes a visual program content feature being correlated to a timeperiod in the timeline of the visual program content in which the visualprogram content feature appears.
 4. The machine or group of machines ofclaim 1, wherein the timeline rule indicates that an ancillary dataobject is to be presented to the audience member when a correlatedvisual program content feature appears in the visual program content. 5.The machine or group of machines of claim 1, wherein the environmentalfeatures of the audience member include: a date range in which theaudience member watches the visual program content; a geographicallocation of the audience member; a time zone of the audience member;gender of the audience member; and an age range of the audience member.6. The machine or group of machines of claim 1, wherein theenvironmental rule indicates that an ancillary data object is to bepresented to the audience member when a correlated environmental featureof the audience member is present.
 7. The machine or group of machinesof claim 1, wherein the indicating that the ancillary data objects areto be presented to the audience member includes making an ancillary dataobject selectable by the audience member on a display on which thevisual program contents is displayed or on a second display, wherein theaudience member selecting the ancillary data object when presenteddisplays additional information regarding the ancillary data object. 8.The machine or group of machines of claim 1, wherein the indicating thatthe ancillary data objects are to be presented to the audience memberincludes making an ancillary data object selectable by the audiencemember on a display on which the visual program contents is displayed oron a second display only when a) two or more correlated visual programcontent features appear in the visual program content during playbackand b) two or more environmental features of the audience member arepresent.
 9. The machine or group of machines of claim 1, wherein theindicating that the ancillary data objects are to be presented to theaudience member includes making an ancillary data object selectable bythe audience member on a display on which the visual program contents isdisplayed or on a second display only when a) one or more correlatedvisual program content features appear in the visual program contentduring playback and b) three or more environmental features of theaudience member are present.
 10. The machine or group of machines ofclaim 1, wherein the timeline rule indicates that an ancillary dataobject is to be presented to the audience member when a) a firstcorrelated visual program content feature appears in the visual programcontent and b) a second correlated visual program content featureappears in the visual program content at the same time as the firstcorrelated visual program content feature.
 11. A method for curatingancillary data to be presented to audience members of a visual programcontent, the method comprising: creating a timeline rule that correlatesancillary data objects to respective visual program content features,the visual program content features correlated to respective instanceson a timeline of the visual program content; creating an environmentalrule to correlate the ancillary data objects to respective environmentalfeatures of an audience member; and indicating that the ancillary dataobjects are to be presented to the audience member when both thetimeline rule and the environmental rule are met such that the ancillarydata objects may be presented to the audience member when both a) therespective ones of the visual program content features appear in thevisual program content during playback by the audience member and b) therespective environmental features are present.
 12. The method of claim11, wherein the visual program content features are selected from agroup comprising: content title; characters; content script or plot;content scene; wardrobe wore by the characters; performers, producers,or directors of the content; items appearing in the content;geographical locations where the content storyline takes place; andmusic in audio of the content.
 13. The method of claim 11, wherein thevisual program content features being correlated to respective instanceson a timeline of the visual program content includes a visual programcontent feature being correlated to a time period in the timeline of thevisual program content in which the visual program content featureappears.
 14. The method of claim 11, wherein the timeline rule indicatesthat an ancillary data object is to be presented to the audience memberwhen a correlated visual program content feature appears in the visualprogram content.
 15. The method of claim 11, wherein the environmentalfeatures of the audience member include: a date range in which theaudience member watches the visual program content; a geographicallocation of the audience member; gender of the audience member; and anage range of the audience member.
 16. The method of claim 11, whereinthe environmental rule indicates that an ancillary data object is to bepresented to the audience member when a correlated environmental featureof the audience member is present.
 17. The method of claim 11, whereinthe indicating that the ancillary data objects are to be presented tothe audience member includes making an ancillary data object selectableby the audience member on a display on which the visual program contentsis displayed or on a second display, wherein the audience memberselecting the ancillary data object when presented displays additionalinformation regarding the ancillary data object.
 18. The method of claim11, wherein the indicating that the ancillary data objects are to bepresented to the audience member includes making an ancillary dataobject selectable by the audience member on a display on which thevisual program contents is displayed or on a second display only when a)two or more correlated visual program content features appear in thevisual program content during playback and b) two or more environmentalfeatures of the audience member are present.
 19. The method of claim 11,wherein the indicating that the ancillary data objects are to bepresented to the audience member includes making an ancillary dataobject selectable by the audience member on a display on which thevisual program contents is displayed or on a second display only when a)one or more correlated visual program content features appear in thevisual program content during playback and b) three or moreenvironmental features of the audience member are present.
 20. Themethod of claim 11, wherein the timeline rule indicates that anancillary data object is to be presented to the audience member when a)a first correlated visual program content feature appears in the visualprogram content and b) a second correlated visual program contentfeature appears in the visual program content at the same time as thefirst correlated visual program content feature.